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METHOD AND SYSTEM FOR CONTROLLING PRESENTATION OF 
INFORMATION TO A USER 
BASED ON THE USER'S CONDITION 



TECHNICAL FIELD 

5 The present invention relates generally to computer program user 

interfaces, and more particularly to presenting information to a user based on the user's 
current condition. 

BACKGROUND OF THE INVENTION 

As computers become increasingly powerful and ubiquitous, users 
,0 increasingly use their computers for a broad vanety of tasks. For example, in addition 
to traditional activities such as running word processing and database applications, users 
increasing use computers as an integral part of their daily lives. Programs to schedule 
activities, generate reminders, and provide rapid communication capabilities are 
becoming increasingly popular. Moreover, computers are increasingly present during 
15 virtuallv all of a person's daily activities. For example, hand-held computer organizers 
(e.g., PDAs) are increasingly common, and communication devices such as portable 
phones are increasingly incorporating computer capabilities. Thus, users may be 
presented with output information from one or more computers at any time. 

While advances in hardware make computers increasingly ubiquitous, 
20 traditional computer programs are not typically designed to efficiently present 
information to users in a wide vanety of environments. For example, most computer 
programs are designed with a prototypical user being seated at a stationary computer 
with a large display device, and with the user devoting full attention to the display. In 
that environment, the computer can safely present information to the user at any time, 
25 with minimal nsk that the user will fail to perceive the information or that the 
information will dtsturb the user in a dangerous manner (e.g.. by startling the user while 
they are using power machinery or by blocking their vision while they are moving wuh 
information sent to a head-mounted display). However, in many other environments 
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toe assump.ions abou, the prototype, user are no, tme, and users thu, may „ ot 
perceive ourpu, infon.alion ( ,,.. fmling [0 „ icon ^ ^ _ ^ 

display device when ,, is hols , ered , or fa „ ing „ ^ ^ ^ ^ 

env.ronmen, or when intensely concerning, Similarly, some user acvities mav 
have a low decree of rn.emrpnodirv (,,, abihry ,o safely in.emrp, ,he user) such rha. 
■he user wou,d prefer rha, rhe presentanon of low-imporrance or of all i„fo m a„o„ he 
deferred, or ,ha t infonnation be presented in a non-imrusive manner. 

In addition to assum.ng that a user ts devottng full attentton , 0 th e 
dtspiay, current computer programs typically assume ,ha, only the user is devoting 
■0 attention to the computer system. Thus, current computer programs are no. concerned 
wtth factors related to the user's envtronment, such as w„e,er other people around the 
user are disturbed by information being presented or whether sensitive mfomtation is 
tnadvertentlv made available to others. ,nstead, current computer programs typically 
assume that the user wi„ req ues, output information on,y a, appropriate times and that 
.5 the user win comro, whether others are privy ,o ourpu, information (e.g., by orienting 
the display accordingly or adjusting speaker volumes). 

However, as computers are increasingly present with users and are 
designed ,o presen, ou^u, infonnation o.her than a, a user's immediate reques. (e . 
reminding .he user of an upcoming appoimmem,, computer programs are increasing 
20 likely ,o present infonnation in a manner ihat interrupts the user (and may "be 
bothersome or dangerous), that may no. be perceived by the user even if highly 
important and urgen, ,ha. may disturb o.hers around the user, and tha. may 
inadvertently reveal sensitive infonnation to others. 

A growing lre „d of using wearable computers will only exacerbare this 
25 problem. Such wearable computers are designed to ac. as constant companions and 
imelhgem assistant ,o a user, thus being available ,o receive input from .he user a. any 
"me and to present ourpu. information ,o the user a, any ,,me. Wearable eompmers are 
typicaHy strapped to ,„e user's body or mourned in a holster, and may include a vanerv 
of bo,h ,npu. and outpu. dev.ces. The close association of wearable computers ,o their 
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results in the wearable computer interacting with the user in virtually any social or 
business situation, and thus the likelihood of inappropriate output behavior increases. 



users 



SUMMARY OF THE INVENTION 

Some embodiments of the present invention provide a method and 
5 svstem for controlling presentation of information to a user based on the user's current 
condition. In particular, the system monitors the user and the user's environment, and 
creates and maintains an updated model of the current condition of the user. The user 
condition can include a variety of condition variables, including abstract concepts such 
as the user's current cognitive load, desired level of privacy for output information, and 
,0 desired scope of audience for output information. The user condition can also include 
condition variables indicating physical characteristics (e.g., deafness) and physically 
observable characteristics (e.g., movement or proximity to another object) of the user. 
Upon receiving output information to be presented to the user (e.g., from an application 
program), the system determines an appropriate output device and an appropriate format 
,5 with which to present the information to the user, and then presents the output 
information. In some embodiments, the system also receives description informat.on 
about the output information that describes relevant factors for determining when and 
how to present the output information (e.g., the importance and urgency of the output 
information, the consequences of the user not receiving or ignoring the output 
20 information, etc.). The system executes in some embodiments on a wearable computer 
having a variety of available output display devices. 

In one embodiment, the system presents output information to a user by 
first receiving information about a modeled characteristic of the user which may include 
a modeled preference of the user for receiving sensitive information, a modeled 
25 indication of a current degree of interruptibility of the user, or a modeled preference of 
the user for an amount of people to perceive information presented by the computer. 
The svstem then selects an output device capable of presenting the output information 
in accordance with the modeled characteristic, and presents the output information on 
the selected output device in accordance with the modeled characteristic. 
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In an alternate embodiment, the system presents information to a user on 
one of multiple available output devices. The system monitors the user to collect 
information about a current state of the user, and then mode.s a current user condition 
based on the collected mformauon by determining a current level of pnvacy desired by 
S the user that indicates a group of people allowed to perceive information presented by 
the computer, by determmmg a current scope of audience destred by the user that 
indicates how many people are intended to perceive mformauon presented by the 
computer, and by determmmg a current cognitive load of the user that indicates ability 
of the user to devote attention to the computer. The system then receives output 
10 mformauon to be presented to the user, and presents the output information in a manner 
cons.stent with the modeled current user condition by selecting one of the output 
devtces such that information presentation capabilities of the selected output device 
support the determined current desired level of pnvacy, the determmed current desired 
scope of audience, and the determmed current cognitive load, and by presenting the 
15 output information to the user on the selected output device, so that the presentation of 
information by the system satisfies the modeled current user condition. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates a user wearing a body-mounted computer executing 
an embodiment of the Condition-Dependent Output Supplier (CDOS) system of the 
20 present invention. 

Figure 2 is a block diagram illustrating the contents and information flow 
of an embodiment of the CDOS system. 

Figure 3 is an illustrative example of a model of a current user condition. 

Figure 4 is an illustrative example of a User Characterization Module. 

Figure 5 is an illustrative example of an Output Device Selector Module. 

Figure 6 is 311 exemplary flow diagram of an embodiment of the User 
Characterization routine. 

Figure 7 is an exemplary flow diagram of an embodiment of the 
Characterize User subroutine. 



25 
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Figure 8 is an exemplary flow diagram of an embodiment of the Output 
Device Selector routine. 

Figure 9 is an exemplary flow diagram of an embodiment of the Format 
And Present Output Information subroutine. 

5 DETAILED DESCRIPTION OF THE INVENTION 

An embodiment of the present invention provides a method and system 
for controlling presentation of information to a user based on the user's current 
condition. In particular, the Condition-Dependent Output Supplier (CDOS) system 
monitors the user and the user's environment, and creates and maintains an updated 

10 model of the current condition of the user. Upon receiving output information to be 
presented to the user (e.g., from an application program), the CDOS system determines 
an appropriate output device and an appropriate format with which to present the 
information to the user, and then presents the output information. In some 
embodiments, the CDOS system also receives description information about the output 

15 information that describes relevant factors for determining when and how to present the 
output information (e.g., the importance and urgency of the output information, the 
consequences of the user not receiving or ignoring the output information, etc.). 

In one embodiment, the model of the user's current condition includes a 
variety of condition variables that represent information about the user and the user's 

20 environment at varying levels of abstraction. For example, information about the user 
at a low level of abstraction can include raw physiological data (e.g., heart rate and 
EKG) and geographic information (e.g., location and speed), while higher levels of 
abstraction may attempt to characterize or predict the user's physical activity (e.g., 
jogging or talking on a phone), emotional state (e.g., angry or puzzled), desired output 

25 behavior for different types of information (e.g.. to present private family information 
so that it is perceivable only to myself and my family members), and cognitive load 
(i.e., the amount of attention required for the user's current activities). Background 
information which changes rarely or not at all can also be included, such as the user's 
age, gender and visual acuity. The model can similarly hold environment information 
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« a low lev,, of abstraction, such as air .etnperarure or raw dala from 

or - higher levels of abstraction ^ ^ -on sensor, 

objects, and locations The moH.ru 'aentit.es of nearby people, 

information added explicit fi^ h C ° nditi °" ^ 

u ex P«citIy from other sources ie e anni.w 
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* trough a head-mourned mtcrophone,. that fact may p ro vide less user condition 
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information to the CDOS system since the user can supply such audio information 
while engaged in a variety of types of physical activity. Those skilled in the art will 
appreciate that there are a wide variety of input devices with which a user can supply 
information to the computer system, including voice recognition devices, traditional 
5 qwerty keyboards, chording keyboards, half qwerty keyboards, dual forearm keyboards, 
chest mounted keyboards, handwriting recognition and digital ink devices, a mouse, a 
track pad, a digital stylus, a finger or glove device to capture user movement, pupil 
tracking devices, a gyropoint, a trackball, a voice grid device, digital cameras (still and 
motion), etc. 

10 In addition to the information received via user input, the User 

Characterization Module also uses sensed information about the user. For example, a 
variety of sensors can provide information about the current physiological state of the 
user, geographical and spatial information (e.g., location and altitude), and current user 
activities. Some devices, such as a microphone, can provide multiple types of 

15 information. For example, if a microphone is available, the microphone can provide 
sensed information related to the user {e.g., detecting that the user is talking, snoring, or 
typing) when not actively being used for user input. Other user-worn body sensors can 
provide a variety of types of information, including that from thermometers, 
sphygmometers, heart rate sensors, shiver response sensors, skin galvanometry sensors, 

20 eyelid blink sensors, pupil dilation detection sensors, EEG and EKG sensors, sensors to 
detect brow furrowing, blood sugar monitors, etc. In addition, sensors elsewhere in the 
near environment can provide information about the user, such as motion detector 
sensors (e.g., whether the user is present and is moving), badge readers, still and video 
cameras (including low light, infra-red, and x-ray), remote microphones, etc. These 

25 sensors can be both passive (i.e., detecting information generated external to the sensor, 
such as a heart beat) or active (i.e., generating a signal to obtain information, such as 
sonar or x-rays). 

Stored background information about the user can also be supplied to the 
User Characterization Module. Such information typically includes information about 
30 the user that changes at most infrequently, although it is possible to frequently update 
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the stored background information to reflect changing conditions. For example, 
background information about the user can include demographic information (e.g., race,' 
gender, age, religion, birthday, etc.) if it can affect how information is presented 'to the 
user. User preferences, either explicitly supplied or learned by the system, can also be 
5 stored as background information. Information about the user's physical or mental 
condition which affects the type of information which the user can perceive, such as 
blindness, deafness, paralysis, or mental incapacitation, is also important background 
information that allows the CDOS system to adapt to the user's capabilities. 

In addition to information related directly to the user, the User 
.0 Characterization Module also rece.ves and uses .nformation related to the environment 
surrounding the user. For example, dev.ces such as microphones or motion sensors 
may be able to detect whether there are other people near the user and whether the user 
is interacting with those people. Sensors can also detect env ir onmental conditions 
which may affect the user, such as air thermometers or geigercounters. Sensors, either 
15 body-mounted or remote, can also provide information related to a wide variety of user 
and environment factors including location, orientation, speed, direction, distance, and 
proximity to other locations GPS and differential GPS devices, orientation 

tracking devices, gyroscopes, altimeters, accelerometers, anemometers, pedometers, 
compasses, laser or optical range finders, depth gauges, sonar, etc.). Identity and 
20 informational sensors (e.g., bar code readers, biometric scanners, laser scanners, OCR, 
badge readers, etc.) and remote sensors (e.g., ' home or car alarm systems, remote 
camera, national weather service web page, a baby monitor, traffic sensors, etc.) can 
also provide relevant environment information. 

In addition to receiving information directly from low-level sensors, the 
25 User Characterization Module can also receive information from devices which 
aggregate low-level information into higher-level data constructs (e.g., face recognizers, 
gesture recognition systems, affective/emotion recognizers, etc.). The user can also 
explicitly supply information about their current condition (e.g., "I have a high 
cognitive load and do not want to be disturbed," "I am dealing with private family 
,0 information that should not be made available to anyone outside my immediate family," 
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or "I am in an environment where I do not wish to disturb others around me"). The 
CDOS system can also receive current date and time information in order to both track 
changes over time and to utilize information such as the user's stored schedule. 
Previously-created models of the user's condition can also be retrieved and used as a 
5 default or to detect changing conditions. The Output Dev.ce Selector Module can also 
supply informatton indicating the types of output currently being presented to the user, 
thus providing information about the user's current activities and cognitive load. 
Finally, other entities (e.g., application programs) can directly prov.de user condition 
informauon (e.g., new condition vanables. whether application-specific or not, or new 

10 values for existing condition variables). 

In some embodiments, CDOS systems communicate between 
themselves, such as via a wireless medium or when cabled together. Tins 
int ercommumcation can occur automatically, or at the instruction of one or more of the 
users of the communing CDOS systems. When multiple CDOS systems 
,5 communicate, a variety of types of informauon can be passed between the CDOS 
systems For example, a first CDOS system receiving information from other CDOS 
systems can use those systems as a type of remote sensor in which information recetved 
by the User Characterization Modules of the other CDOS systems is also supplied as 
input to the User Characterization Module of the first CDOS system. Other CDOS 
20 systems may have access to information about the surrounding environment (e.g., a 
digital camera) that the first CDOS system does not have. Alternately, informauon 
about the users of the CDOS systems can be exchanged to facilitate further 
communication between the CDOS systems or between the users (e.g., nottfymg one 
user that another user has a htgh cognitive load and does not wish to be disturbed). 
25 Multiple CDOS systems can also act as cooperative systems in which one or more 
users' CDOS systems are shared with other users (e.g., making available excess 
computing power or the use of an output device). 

After the User Characterization Module receives informauon about the 
user and the surrounding environment from one or more information sources, the User 
30 Characterization Module will use this informauon to create a current model of the 
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user's condit.cn. In one embodiment, the User Characterization Module merely stores 
the data it receives (even when it is at a low-level of abstraction) and then allows the 
Output Device Selector Module or apphcation programs to directly use that stored 
.nformauon when making decisis related to presenting output information. In an 
5 alternate embodiment, the User Charactenzat.on Module uses received low-level data to 
generate higher-leve, representations of the user's observable activities (e.g., walkin* 
watching a movie in a movie theater, talk.ng to coworkers at the office, etc.). 

In yet another embodiment, the User Characterization Module further 
characterizes the user's condition with respect to condition variables that are not 
.0 directly observable. Such condition variables include the current cognitive .oad of the 
user (indicating amount of attention required for the user's current activities and thus 
the ab.lity of the user to devote attention to the computer), the current degree of 
mterruptibility for the user (indicating ability to safely interrupt the user), the current 
degree of intrusiveness of output on the environment (indicating impact of output on the 
15 surrounding environment), the user's desired scope of audience for information bein* 
output (indicating how many people should be able to perceive the information) the 
user's desired level of privacy for information being output (indicating the group of 
people who are allowed to perceive the information), and the user's desired level of 
solitude (indicating the user's current desire to avoid intrusions). 
■0 User condition variables can also represent abstract principles about the 

user and the surrounding environment, such as the user's relationship to other objects, 
people, or locations (e.g., being at their desk, being in their office, being near the drug 
store, talking to a particular person, etc.). In some embodiments, CDOS systems can 
supply information about user condition variables and their values to other CDOS 
5 systems, and those other CDOS systems can add the user condition variables and/or 
values to their model of their user condition if appropriate (e.g., ambient air 
temperature, or an emotional state of a CDOS system's user that is sensed by another 
CDOS system). 

The values for the user condition variables can directlv impact how and 
when output informat.on should be presented. For example, when the cognitive load is 
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high or the degree of interruptib.liry is low, output information may be presented on a 
tactile display using light pressure so that the presentation is minimally intnisive on the 
user. Alternately, the presentat.on of the information may be deferred if no appropriate 
output device is available or if interrupting the user is not warranted by low-importance 
5 or low-urgency information. When the output information is sens.tive and others 
present are not included in the current desired level of privacy, the information may be 
presented on an eyeglass-mounted display, or the information may be presented via an 
earpiece speaker when the scope of audience or intrusiveness on the surrounding 
environment dictates that others not perceive the presented information. Finally, if the 
l0 user's desired level of solitude indicates that the user does not want to receive output 
information {e.g., while asleep, in the bathroom, involved in an intimate activity, etc.), 
presentation of all output information or of all but highly urgent and important output 

information may be deferred. 

Those skilled in the art will appreciate that the User Characterization 
15 Module may receive contradictory information related to one or more aspects of the 
user condition. For example, a motion sensor device may indicate that no one else is 
present in a room, while a speech recognizer may report that another person is present. 
Mediation of such contradictory data can be handled in a variety of ways. For example, 
it may be possible to reconcile such data (e.g., the user is communicating with another 
,o person via a telephone with a loudspeaker). Alternately, the data could reflect different 
readings for changing conditions (e.g., ambient air temperature may have changed 
quickly after a window was opened). Alternately, when data truly conflicts, it may be 
impossible to reach a conclusion about a user condition variable, or the value of the 
variable may be represented as having varying degrees of uncertainty or belief. Both 
25 particular information sources (e.g., sensors) and particular pieces of input information 
can be categorized as to their quality and reliability to assist with mediation or to better 
model the user condition. In addition, input information can be time-stamped and 
otherwise identified to assist the User Characterization Module. 

Those skilled in the art will also appreciate that a variety of factors can 
30 influence the determination of values for each of the condition variables, and that the 
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values for the variables can be stored in a vanety of ways (e.g., a number on a scale of 
1-100 or 0-255, a probability distribution, a value from a delimited set of possibilities, a 
fuzzy logic value, etc.). Factors which can affect the cognitive load of a user include if 
the user is talking (and the volume of the speech), is talking on the phone, physical 
5 movement such as walking or dnving, being stationary, being seated and stationary, 
ambient l:,.u and sound, stress and hunger levels, a level of rest (e.g., a low level due to 
a recent lack of sleep), activity such as reading e-mail or riding a bull, historical data 
(e.g., user has low threshold for cognitive load while watching baseball games), a 
physical or mental disability, location at home or therapist's office), presence and 
10 frequency of user input such as keyboard or mouse activity, presentation of output 
information to the user, emotional state, explicit indications from user, etc. Similarly, 
factors that can affect desired level of privacy and desired scope of audience include the 
identity of others near the user, the proximity of others to the user, explicit tagging of 
activities or information (e.g., email in my personal account is private for only me, 
15 while email in my family account is private for family members), nature of work being 
performed (e.g., balancing a checkbook, playing a computer game, or revising a 
business spreadsheet), location, historical data, explicit indications from user, etc. 

Those skilled in the an will appreciate that when information is being 
processed and shared between multiple systems, it is necessary for a context to be 
20 shared so that a semantic understanding of what is represented by information can be 
conveyed. For example, merely reporting data for air temperature as being 50 is 
insufficient. Not only is it unclear what scale is being used (e.g., Fahrenheit or Celsius), 
it is also unclear exactly what information is being represented (e.g., the air surrounding 
the user inside a heated room, or the outside air). Thus, the components of the CDOS 
25 system have a shared context as to the meaning of input information and user condition 
variables, including having consistency among the components generating values of the 
user condition variables (e.g., rules in the User Characterization Module) and those 
using the generated values (e.g., the Output Device Selector Module and Format 
Modules). In addition, when information from the CDOS system (e.g., user condition 
variables and their values) is shared with other entities (e.g., other CDOS systems or 
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application programs with output information to be presented), shanng of the context 
with these other entities enables the information to be useful. In some embodiments, 
other systems are deigned to share the same context (e.g., via a published API), while 
in other embodiments additional information descnbing the shared informauon can be 
5 supplied along with the shared information to establish a shared context. 

In addition to there being a wide variety of factors which can affect 
vanous condition variables, the User Characterization Module can generate the model 
of the current user condition in a variety - of ways. In one embodiment, the User 
Characterization Module includes a variety of rules such that when a test portion of the 
l0 rule is currently true, then the result portion of the rule is activated or performed (e.g., to 
cause the value of a condition variable to be modified or to satisfy the test portion of 
other rules). For example, a rule could indicate that if the user is talking or the 
surrounding environment is loud, then non-auditory output is preferable to auditory 
output. When this first rule was satisfied, the result could trigger the satisfaction of a 
15 second rule, such as a rule stating that while non-auditory output is currently preferable 
then an eyeglass-mounted display device will be used for output. Alternately, a second 
rule could state that although non-auditory output is currently preferable, an earpiece 
speaker device will be used for highly sensitive information. Another example of a rule 
is that if the user is driving an automobile at high speed in bad weather conditions, it is 
20 likely that the user has a high cognitive load and would be unable to safely devote much 

attention to the computer system. 

Those skilled in the an will appreciate that there are a var,e.y of 
techniques for combining different types of input mformation and processing « to 
generating output .nformation, including look-up tables, neural networks, expert 
25 system, genettc algorithm, probabilistic belief networks, etc. In addition, values for 
some user condition variables may be calculated only periodically or only upon specfic 
request for the value (e.g., computationally intensive vanable values such as those 
generated by a face recogmzer), even if.be appropnare input information is supplted 
more frequently. Conversely, some embodtments of the CDOS system may allow .he 
30 User Characrerization Module to request or retneve the appropnate inpu, informauon 
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needed to calculate one or more user condition vanables. thus performing demand- 
dnven processing. An il.ustrative example of a User Characterization Module ,3 
described in greater detail with respect to Figure 4. 

In some embodiments, CDOS systems can supply , 0 other CDOS 
systems varions information related ,0 generating the mode, of the user condition and 
those other CDOS systems can use that mode, generanon information in addition ,0 or 
m piace of their own rao d=, generanon informatton. For examp.e, if m,es are being 
used ,0 generate the mode, of the user condition, one CDOS system can supply some or 
all of its rules to other CDOS systems. Similarly, default and/or specialized sets of 
.» mode, generation information can be supplied ,0 a CDOS system, etther from other 
CDOS systems or by .oading mat infotrnat.on onto the CDOS system. A default se, of 
tu.es may be used by CDOS system until leannng by the system adds or modifies the 
default rules to better mode, the user of the system. Simiiar.y, other programs (. . 
apphcatton programs) can supply rules to the CDOS system, such as rules specific to' 
15 that appfication program. Various spectalized sets of ru.es can also be supplied For 
example, sets ofm.es may be spectalized based on occupation (e.g., a muse, a secretarv 
a field technician, or a firefighter), gender (e.g., a woman's rule* may understand 
Phystologica, symptoms related ,0 pregnancy or other female-spectfic conditions), a„e 
or any of a variely of other specialization types. 

After the User Characterization Module has created a model of the user's 
current condition, the Output Device Selector Module and the one or more Forma, 
Modules can then use the mode. ,o determine when and how ,0 present output 
tnformauon ,0 the user. The Output Deyice Selector Module firs, recetves output 
tnformatton ,0 be presented ,„ the user (e.g., from an application program), and .hen 
S selects an appropriate output device for presenting the information .0 the user. For 
example, if the model of the user condition indicates that auditory output is currently 
preferable ,0 other forms of output and the output information can be presented andtblv 
then the Output Dev.ce Selector Module selects an output device that supports audible 
output. Alternately, the value of a desired level of privacy, des.red scope of audience 
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or current cognitive load user condition variable may indicate that audible output is 
currently preferable. 

While information to be presented to the user will often be generated by 
an entity outside the CDOS system, the CDOS system may also generate information to 

5 be presented to the user (e.g., an indication of low battery power, or of an error when 
adding a rule to the User Characterization Module). In addition, in some embodiments 
external entities such as an application program can directly access the model of the 
user condition and make their own determination as to when, where and how to present 
output information (i.e., bypassing the Output Device Selector Module and/or the 

10 Format Modules). Thus, if the modeled user condition indicates that particular output 
information should not currently be presented to a user, the external entity can postpone 
or cancel the presentation of the output information without ever supplying the output 
information to the CDOS system. It may also be possible to configure the CDOS 
system to automatically notify the external entities of the values of one or more user 

15 condition variables, such as by pushing that information to the external entities when 
changes occur in the values or by periodically notifying the external entities of the 
current values. 

If the selected output device supports different ways to audibly present 
information (e.g., different audio levels or a choice of speakers) or the ability to output 

20 information to multiple senses, the Format Module for the output device will then 
format the output information appropriately based on the model of the user condition 
(e.g., lowering the volume to minimize the degree of interruptibility for non-urgent 
information or to prevent non-employees of the business from perceiving sensitive 
business information, or using 3-D sound that is output as if from a sound source 

25 located at a particular location in the user's surrounding environment). Alternately, if 
the current model of the user condition indicates that output to the user is currently 
inadvisable (e.g., the user has a high cognitive load, is asleep, or sensitive information 
cannot be safely presented in the current environment), the Output Device Selector 
Module instead defers or denies the presentation of the information. 
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In one embodiment, the Output Device Selector iModule selects output 
devices by first characterizing each of the output devices relative to selected condition 
variables, such as cognitive load, desired level of privacy, desired scope of audience, 
and intrusiveness on the environment. For example, an eyeglass-mounted display may 
5 have a high rating for ability to present sensitive information to only the user, but may 
have a low rating for lack of intrusiveness on the user (particularly if the user has a high 
cognitive load from another visual activity). Similarly, an olfactory device which can 
output various smells may be low on the intrusiveness scale, but may be useful for 
presenting only limited types of output (e.g., a soothing aroma when the user has high 
10 blood pressure and a high pulse). Output devices can also be characterized on the basis 
of the user sense (e.g., olfactory or visual) to which the output information will be 
presented. 

After the output devices have been characterized on the basis of the 
condition variables, the Output Device Selector iModule then selects the one or more 
15 output device which are most appropriate for the user's current condition and for the 
information to be output. In some situations, a characterization of a device relative to a 
condition variable is dependent on the circumstances rather than being inherent in the 
capabilities of a device. For example, a stereo or a television may have a high degree of 
privacy while only the user is in the room, but the characterization for these devices 

20 may change to a low degree of privacy when others enter the room. In some 
embodiments, such devices are represented with a characterization that is a range of 
values, with only a single value or a subset of the range selected at a given time based 
on the current circumstances. 

In addition to supplying the output information to be presented, an 

25 external entity can also supply information that describes the output information, such 
as the relative importance and urgency (i.e., the degree of deferability, such as time 
sensitivity) of the information, as well as the consequences of ignoring the information. 
In the same manner that the output devices can be characterized relative to condition 
variables, they can also be characterized relative to such factors in the description 

30 information. For example, an eyeglass-mounted display and an earpiece speaker with 
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adjustable volume may both be highly-rated with respect to their ability to present 
important information that has a high consequence of be.ng ignored. The earpiece 
speaker may have a wide range of ratings for these factors, however, since it is also able 
to present low importance information {e.g., at a low audio volume which can be easily 
5 ignored by the user if the user so chooses). Conversely, the eyeglass-mounted display 
may not be able to unobtrusively present visual information, and thus may have a small 
range of ratings for this factor. Thus, after the Output Device Selector Module receives 
the information to be output and optionally receives a description of the information, 
the Output Device Selector Module then uses the model of the user condition to 
l0 determine which output device (if any) to use to present the information to the user, and 
a corresponding Format Module for that device determines the appropriate format with 
which to present the information to the user. 

In one embodiment, the Output Device Selector Module includes a 
characterization of each output device available to the CDOS system relative to the user 
,5 condition variables of cognitive load, desired level of privacy, desired scope of 
audience, and desired level of intrusiveness on others, as well as to output information 
description factors of relative level of importance, deferability, and consequence of 
ignoring. The one or more devices which best match the current user condition and the 
current output information will be selected, including using user preferences to select 
20 between different devices. Those skilled in the art will appreciate that the Output 
Device Selector Module could determine an appropriate output device in a variety of 
other ways, including receiving a direct specification from the entity supplying the 
output information, selecting the device with the widest range of capabilities relative to 
the type of information to be output, etc. In addition, a defined API (applicatton 
25 program interface) can be designed between external entities such as application 
programs and the CDOS system. The defined API will allow application programs to 
supply information to User Characterization Modules, extract and add information to 
the model of the user condition, and supply output information and description 
information to Output Device Selector Modules. An illustrative example of an Output 
30 Device Selector Module is described in greater detail with respect to Figure 5. 
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printer 166, either directly or via the non-portable computer, and directly to the 
telephone 168. As the user moves out of range of the remote input and output devices, 
the CDOS system will be updated to reflect that the remote output devices are not 
currently available to receive output. 
5 Those skilled in the art will appreciate that computer systems 120 and 

150, as well as their various input and output devices, are merely illustrative and are not 
intended to limit the scope of the present invention. The computer systems may contain 
additional components or may lack some illustrated components. For example, it is 
possible that the CDOS system could be executed on the non-portable computer, with 
10 the body-mounted computer replaced by a thin client such as a transmitter/receiver for 
relaying information between the body-mounted input and output devices and the non- 
portable computer. Alternately, no devices or computers may be worn by the user. 

In addition, the body-mounted computer may be connected to one or 
more networks of other devices through wired or wireless communication means (e.g., 
15 wireless RF, a cellular phone or modem, infrared, physical cable, a docking station, 
etc.), either with or without support from other computers such as the computer 150. 
For example, the body-mounted computer of a user could make use of output devices in 
a smart room, such as a television and stereo when the user is at home, if the body- 
mounted computer can transmit information to those devices via a wireless medium or 
20 if a cabled or docking mechanism is available to transmit the information. Alternately, 
kiosks or other information devices can be installed at various locations (e.g., in airports 
or at tourist spots) to transmit relevant information to body-mounted computers within 
the range of the information device. 

Those skilled in the art will also appreciate that specialized versions of 
25 the body-mounted computer and CDOS system can be created for a variety of purposes. 
For example, a wearable defibrillator could be created such that the CDOS system 
monitors the user's physiological condition to determine whether output electrical 
energy needs to be supplied to the user, and to automatically supply such output energy 
when needed. Alternately, a breathing monitor system could monitor the user and take 
30 appropriate action if a breathing problem develops (e.g., calling 911 or notifying a 
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nearby medical care provider), or a sleep-sensing system for a dnver of a vehicle could 
simulate the driver (e.g., audibly or tactilelv) when the dnver becomes drowsy A 
variety of other physiological conditions can be similarly monitored, and other 
specialized versions of the system can simtlarly be implemented (e.g., an aid for a deaf 
> person that performs speech recognition on spoken words in order to v 1S ually display 
the words, or a mood enhancing device that triggers various soothing environmental 
output in response to a user condition such as stress or anger). 

Figure 2 illustrates an embodiment of the body-mounted computer P0 
in greater detail. The computer includes a memory 270, a CPU 280, and a storage 
device 290. The CDOS 100 system is executing in memory, as well as one or more 
distinct application programs 260. As the body-mounted computer receives various 
input information, the informat.on is forwarded to the User Characterization Module 
205 of the CDOS system. These types of information include explicit user input to the 
computer, sensed user information, and sensed environment information. The User 
Characterization Module can also receive date and time information from the CPU or 
from some other source, and can retrieve stored information (e.g., user preferences, 
definitions of various user-defined groups, or a default model of the user condition) 
from the storage device. It is also possible for one or more of the application programs 
to optionally supply application-supplied information 265 to the User Characterization 
Module. This information can include any type of user condition information to which 
the application program has access, such as user location or physiological state. In 
addition, the application programs can create new user condition variables (e.g., an 
indication of where the user's pupil is directed for an interactive game program), 
including those to be used only by that application program. Similarly, a utility 
program could supply user condition information that is useful to a specified subset of 
application programs (e.g., to various application programs from a single vendor or of a 
certain type). 

After the User Characterization Module receives one or more of these 
types of information, it processes the information and creates a model of the user 
condition 210 which will include multiple user condition variables (with current values 
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for some or all of the variables). After the model of the user condition has been created, 
the current model will be made available to the User Characterization Module to assist 
in the characterization of the user, such as with changes over time. The model will also 
be available to the Output Device Selector Module 215 to assist with presentation of 

5 output information. Moreover, the model of the user condition can additionally be 
stored in a permanent manner, such as on the storage device, if non-current versions of 
the user condition model are useful. Similarly, the User Characterization Module, 
Output Device Selector Module, and any Format Modules can be permanently stored 
before being executed in memory, and any changes made to the modules while they are 

10 executing can also be saved. 

When the Output Device Selector Module receives output information to 
be presented, such as from one of the application programs, it uses the current model of 
the user condition as well as information about the available output devices to 
determine an appropriate output device on which to present the information to the user. 

15 In some embodiments, the Output Device Selector Module may retrieve information 
about the output device characteristics upon initialization, such as from the storage 
device. Alternately, the Output Device Selector Module could instead receive the 
information directly from the output devices as they are dynamically configured. The 
source of the output information can also supply a description of the information to 

20 assist in selecting where, when and how to present the information to the user. After an 
output device has been selected, the Output Device Selector Module forwards the 
output information as well as appropriate output information description factors and 
user condition variables to the Format Module for the output device. In the illustrated 
embodiment, Format Modules 220 through 22S correspond to the output devices as 

25 shown. 

When a Format Module receives the output information, it formats the 
output information for presentation, with the formatting based in part on the information 
presentation capabilities of the output device. For example, the output device may be 
able to output information to more than one user sense, in more than one way, and with 
30 varying degrees of amplitude or style (e.g. . flashing text or enlarged icons). The Format 
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Module selects an appropriate method of formatting the information, such as to present 
the information to only the appropriate audience or with the appropriate level of 
intnisiveness, and then sends the information to its corresponding output device for 
display. The Output Device Selector Module will also inform the User Characterization 
5 Module when output is to take place so that the model of the user condition can be 
updated accordingly. 

Those skilled in the art will appreciate that the Format Modules may 
communicate with their corresponding output devices in a variety of ways, and that the 
body-mounted computer in the CDOS system may contain additional components or 
10 may lack some illustrated components. For example, there may not be a one-to-one 
mapping between Format Modules and output devices, functionality performed by the 
Output Device Selector Module and Format Modules may be incorporated together, and 
the creation of the model of the user condition may be performed by a different system 
than that which uses the information to present output information. There may also be 
15 multiple User Characterization or Output Device Selector Modules, such as one User 
Characterization Module for each relevant high-level condition variable. Alternately, 
external entities such as the application programs could add their own User 
Characterization, Output Device Selector or Format Modules, or could directly access 
the model of the user condition in order to perform presentation of output information. 
20 Accordingly, the present invention may be practiced with other computer system 
configurations. 

Figure 3 is an illustrative example of a Model of User Condition 210. As 
is shown, the model reflects the condition of user X at time 14:22 hours on the 
displayed date. The illustrative model of the user condition includes a variety of user 
25 condition variables at different levels of abstraction, including low-level information 
directly from user sensors as well as higher-level abstract variables with characterized 
values that reflect a user's current physical and mental states. Historical and time- 
sensitive information can also be included, as shown by the variable illustrating the last 
user input performed by user X. 
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Intermediate- level variables included in the model can be calculated 
from low-level input information such as sensor values. For example, the speed of the 
user could be calculated directly by a sensor such as a pedometer, or could be calculated 
indirectly via information over time from a GPS sensor. In addition, the Speed variable 

5 indicates that additional information can be included in the user model for each 
variable. In the case of the Speed variable, uncertainty about the exact value of the 
variable is demonstrated. Other calculated condition variables include an indication that 
the user is located in their office, is near their desk, and that there are no other people 
physically nearby. These factors can be determined in a variety of ways, such as via a 

10 motion sensor device located on the desk that is tracking the user and the absence of 
other individuals, or by the lack of any sounds from any other people via one or more 
microphones. 

Higher-level condition variables can also be calculated, such as the 
user's current physical activities, the current user cognitive load, the desired level of 
15 privacy, and the desired scope of audience. Information from the microphone or 
directly from the cellular phone could indicate that the user is currently talking on their 
cellular phone, and the speed and motion sensor data could indicate that the user is 
walking. Since the user remains near his desk even though he is walking, the system 
can deduce that the user is pacing about his office or is walking on a treadmill (not 
20 shown). The User Activity variable demonstrates that variables can have multiple 
values, and that information such as a degree of belief or certainty in the value for a 
variable can be added and used by the system. 

The Cognitive Load variable indicates a score of 77 out of 100, thus 
indicating a relatively high cognitive load due to the combination of the user walking 
25 and talking on the phone. Since it is unlikely that information presented by the system 
will be desired to be perceptible by the person on the other end of the phone, the desired 
Scope Of Audience variable indicates that only the user is currently appropriate to 
receive output information. Since the User Characterization Module was able to 
identify the other person on the phone as Doug Smith, an executive level colleague at 
30 user X's company (e.g., by voice recognition or the use of that person's name), the 
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deseed Level Of Pnvacy variable indicates that if information is presented in a manner 
such that the other person can receive i, (e.g., through an external speaker) General 
.nformation about the company as well as executive-level sens.tive information "can be 
presented. Note that although low-level sensors such as a motion detector mav have 
•ndtcated that there are no other people physically nearby, when it was determined that 
the user was talking on a phone, additional information was added to the Nearbv Peop.e 
vanable to ind.cate that someone is within audio perception of the user. 

The remaining displayed portions of the user condition model indicate 
that user preference information and externally supplied information can be included in 
the user condition model. For example, the Application X-Factor 1 vanable has been 
supplied by application X, as well as a value for the vanable. In this case, the value is a 
normal probability distribution with a mean of 23 and a standard deviation of 3. In 
addition, previously supplied user preference information could tndicate which output 
devices and which output formats are preferred by the user. Alternately, the svstem 
could have automatically learned these preferences over time by observing user 
reactions to various outputs, as well as from explicit suggestions and overrides bv the 
user. Those skilled in the an will appreciate that the illustrated user condition model is 
merely illustrative and is not intended to limtt the scope of the present invention. The 
model may contain additional variables or may lack some illustrated vanables. or may 
20 be represented without explicit condition variables at all. 

Figure 4 is an illustrative example of User Characterization Module 205. 
As is shown, the illustrated User Charactenzation Module is for user X and it includes a 
variety of IF-THEN rules. User condition variables are shown with angle brackets 
surrounding them, with some user condition variables (e.g., Speakerphone.Status) not 
shown in the illustrative model of user condition 210 in Figure 3. In addition to the IF- 
THEN rules, WHILE-THEN rules are also shown, as well as an application-specific 
rule (i.e.. the APPX: rule) added by an external application. The illustrative User 
Characterization Module also indicates that the results ponion of the rules (shown after 
the THEN statements) can set or modify the values of condition vanables. such as by 
30 absolute or percentage numerical amounts, and can indicate degrees of belief or 
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uncertainty in values. Groups of people are shown in square brackets (e.g., Company 
Executives), and asterisks are wild-card characters that can match any information. 

As mentioned previously, receiving input related to one user condition 
variable can cause multiple changes to propagate through the set of rules. For example, 
5 if input is received that indicates that the user condition variable 
Desktop. Motion. Sensor.Human.Movement is true and the User Activity variable value 
indicates that user is seated, one of the rules shown indicates that the Nearby People 
variable will be modified (if necessary) to indicate that an "Unidentified Person" is 
physically nearby. Modifying the Nearby People variable can then affect the Level Of 
10 Privacy or Scope Of Audience user condition variables as shown by other rules. Those 
skilled in the an will appreciate that the illustrated User Characterization Module is 
merely illustrative and is not intended to limit the scope of the present invention. The 
model may contain additional rules, may lack some illustrated rules, or may be 
implemented without using rules at all. In addition, the test and/or results portions of 
15 rules can be implemented as invokable functions, including those provided by external 
entities such as application programs. 

Figure 5 is an illustrative example of Output Device Selector Module 
215. As is shown, the module is for user X and it maps each available output device to 
ratings for selected user condition variables and output information description factors. 
20 As is shown, some output devices which are available at times (e.g., pager 502, cellular 
telephone 504, and car radio 506) are not currently available. In addition, earpiece 
speaker 132 may not currently be able to receive output information if it is already in 
use (e.g., the user is listening to music). Alternately, new output information could 
preempt the current use of the earpiece speaker if necessary, or could instead share the 
25 use of the output device (e.g., outputting music to one ear and other information to the 
other ear if the earpiece speaker is part of headphones). 

As is shown, the various output devices are rated with single values or a 
range of values for each factor. While textual values are used, those skilled in the an 
will appreciate that numerical or other types of rating systems could be used. In the 
30 illustrated embodiment, ranges may illustrate the device capabilities in different 
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situations, with the ranges bang restncted in any particular situation. For example the 
earp,ece speaker can accommodate when the user has a very low cognitive load bv 
adjusting the volume to be shght.y louder than the amb.ent environment. Alternated 
even ,f the user has a high cognitive load, the earp.ece speaker can interrupt the user 'if 
> necessary for urgent information by us.ng very loud volumes or distinctive tones In 
addition, the ratings can be adjusted to reflect the specific situation of this user For 
example, since the speaker ,62 is located on the user's desk at work and other 
employees can frequently or always hear the speaker, the value for the desired Level Of 
Privacy may indicate that only business information be presented via the speaker 
) Alternately, the system could present information by sending it to the ce.lu.ar telephone 
if the mformauon is highly sensitive or it is im ponant to interrupt the user. However if 
others are present around the user, frequent use of the cellular telephone can be hi»h,v 
mtnwve to them (particularly in environments such as a lecture or a movie). 

Those skilled in the art will appreciate that the illustrated Output Device 
Selector Module is merely illustrative and is not intended to limit the scope of the 
present invention. The module may contain additional user condition variables and 
output information description factors, may lack some illustrated user condition 
variables and output information description factors, or may select output devices on 
which to present output information in an entirely different manner. In addition some 
embodiments of the Output Device Selector Module may include specific logic, such as 
IF-THEN rules, to be used in conjunction with the mapping of output devices as shown. 

Figure 6 is an exemplary flow diagram of an embodiment of the User 
Characterization Routine 600. The User Characterization Routine retrieves stored 
information related to the user, including a default model of the user condition, receives 
vanou, types of input information related to the user or the user's environment, updates 
the model of the user condition to reflect the new information, and periodically updates 
the model if no information has been received within a prescribed time. The routine 
begins at step 605 where stored information for the user ,s retrieved, including a set of 
characterization rules to be used by the routine. The routine then creates a default 
model of the user condition, such as directly from stored information or by applv.n. the 
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- characterization rules to default user information that is available. The routine then 
continues to step 610 to set a timer, and continues to step 615 to either receive input 
information or to receive a notification that the timer has expired. 

The routine continues to step 622 to determine if input information was 
5 received. If so, the routine continues to step 625 to determine if the information 
received was information input to the computer by the user. If so, the routine continues 
to step 630 to determine if the user input indicates that the user condition should be 
modified, such as by setting a user preference or explicitly changing the value of a user 
condition variable. If so, the routine continues to step 635 to satisfy the user request, 
10 and then returns to step 610. If it was instead determined in step 630 that the user input 
was not directed to the User Characterization Module, the routine continues to step 640 
to forward the user input information to the appropriate destination (e.g., an application 
program). 

After step 640, or if it was determined in step 620 that the timer had 

15 expired or in step 625 that the information received was not user input, the routine 
continues to step 645 to execute the Characterize User Subroutine to update the model 
of the current user condition. After step 645, the routine continues to step 652 to store 
the updated user condition model, and then continues to step 655 to update the 
characterization rules if necessary. The characterization rules can be updated in a 

20 variety of situations, such as if an external entity (e.g., an application) explicitly adds a 
new characterization rule or if the routine monitors the user's behavior and reactions in 
an attempt to learn more appropriate characterization rules. After step 655, the routine 
continues to step 660 to determine if there is more information to receive. If so. the 
routine returns to step 610, and if not, the routine ends at step 695. 

25 Figure 7 is an exemplary flow diagram of an embodiment of the 

Characterize User Subroutine 645. The subroutine is executed when information is 
received related to the user or the user's environment, or when a timer has expired 
indicating that no information has been received for a specified period of time. When 
no information has been received for a period of time, the model of the user's current 

30 condition may need to be updated so that time-sensitive information can be updated in 
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the model of the user condition. The subroutine begins at step 705 where the current 
mode! of the user condition is retrieved. The subroutine then continues to step 710 to 
retneve the current date and time. In step 715. the subroutine determ.nes whether 
information was received or if the timer has expired. If the timer has expired, the 
5 subroutine continues to step 720 to examiner user condition vanables hat represent 
time-sensitive information or historical data and updates them if necessary. The 
subroutine then continues to step 725 to determine if the current date and time trigger 
any characterization rules, and if so, the changes from these triggered rules are 
propagated through the set of rules. 
10 If it was instead determined in step 715 that information to be processed 

was received, the subroutine continues to step 730 to determine if a new user condition 
variable has been defined, such as by an application program, and if so continues to step 
750. If a new user condition variable has not been defined, however, the subroutine 
continues to step 735 to determine if a new user characterization rule is being added. 
1 5 such as by an application program. If so, the subroutine continues to step 740 to add the 
new characterization rule, determine if current values for any user condition variables 
trigger the rule, and if so propagates any changes from the triggered rules through the 
set of rules. If it was instead determined in step 735 that a new characterization rule is 
not being defined, the subroutine continues to step 745 to determine if the current input 
20 information or the current date and time trigger any rules, and if so, changes from those 
triggered rules are propagated throughout the set of rules. In addition to information 
received directly from the user, sensors, or application programs, this input information 
can also be a notification from the Output Device Selector that indicates output 
information is currently being presented to the user. 
25 Al * er steps 725, 730, 740, or 745. the subroutine continues to step 750 to 

store any changes in user condition variables and their values, as well as the new date 
and time, in an updated model of the user condition. The subroutine then continues to 
step 795 and returns. Those skilled in the an will appreciate that a variety of types of 
information related to the user and the user's environment can be received, and that the 
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User Characterization Routine and the Characterize User Subroutine can process this 
information in a variety of ways, including other than with sets of IF-THEN rules. 

Figure 8 is an exemplary flow diagram of an embouiment of the Output 
Device Selector Routine 800. The Output Device Selector Routine receives output 
5 information to be presented to the user, retrieves current characterized information 
about the user from the model of the user condition, determines whether the output 
information should be presented to the user at this time and if so on which output device 
and in what format, and then notifies the User Characterization Module when output 
information is presented to the user. The routine begins at step 805 where output 
10 information to be presented to the user is received or an indication that a timer has 
expired occurs. The routine continues to step 810 to determine if a timer has expired. 
When output information cannot be currently presented to the user (e.g., no satisfactory 
output device is available or presentation to the user could be dangerous or 
inappropriate), the presentation is deferred and a timer is set to indicate when to review 
15 presenting the information. Thus, if it is determined in step 810 that a timer has 
expired, the routine continues to step 815 to retrieve the deferred output information for 
the timer, as well as any description information for the deferred output information. If 
it is instead determined in step 810 that new output information to be presented has 
been r eceived, the routine continues to step 820 where description information for the 
20 output information is optionally received. 

After steps 815 or 820, the routine continues to step 825 to retrieve 
relevant information from the current model of the user condition. The routine then 
continues to step 830 to determine whether to currently present the output information 
to the user. In the illustrated embodiment, this determination is made using the user 
25 condition variables of cognitive load, desired level of privacy, and desired scope of 
audience. In addition, available description information which indicates the importance 
and the deferability of the output information and the consequences of the user ignoring 
or not receiving the output information are considered, as is any user preference 
information. Current values for these user condition variables and description factors. 
30 as well as whether available output devices can support the necessary formatting of 
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information (e.g., presenting information to the appropriate scope of audience or at the 
appropriate level of intrusiveness for the user s cognitive load), are thus used in the 
determination. Those skilled in the an will appreciate that other factors can be used for 
this determination or that the determination could be made in other ways. 

The routine then continues to step 835 to determine whether the 
presentation of the information is to be deferred or not. If the presentation is to be 
deferred, the routine continues to step 840 to store the output information as well as its 
description information, and to set a timer for the information at which time the 
presentation of the output information will be reconsidered. If it is instead determined 
in step 835 that the information presentation is not to be deferred, the routine continues 
to step 845 where an available output device is selected. In the illustrated embodiment, 
the output device whose information display capabilities and ratings best match the user 
condition variables and description information factors of interest is chosen. The 
routine then continues to step 850 to execute the Format And Present Output 
Information Subroutine, and then continues to step 855 to notify the User 
Characterization Module of the presentation of the output information. After step 840 
or step 855, the routine continues to step 860 to determine if there are currently timers 
set or there is more output information to be received. If so, the routine returns to step 
805, and if not the routine ends at step 895. 
20 Figure 9 is an exemplary flow diagram of an embodiment of the Format 

And Present Output Information Subroutine 850. The subroutine receives output 
information to be presented and its description information, receives relevant user 
condition variables and user preference information, selects a user sense to which the 
output information will be presented (if the output device supports more than one). 
25 selects appropriate formatting with which to present the output information, and 
presents the output information to the user. The subroutine begins at step 905 where 
output information is received, as well as the description information factors, user 
condition variables, and relevant user preference information. The subroutine continues 
at step 910 to select a user sense that is supported by the selected output device. 
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In step 915, the subroutine selects formatting for the output information 
that is appropriate for the user condition variables. O'. nut information description, and 
user preferences. Those skilled in the an will appreciate the formatting of the output 
information will vary with each user sense (e.g., adjusting volume for the audio sense 
5 and adjusting pressure for the tactile sense), as well as with the specific output device. 
After the formatting for the output information is selected, the subroutine continues to 
step 920 to present the output information to the user with the selected formatting. If 
the Scope Of Audience and Level of Privacy user condition variables indicate that the 
information can be presented to other people currently present and the selected output 
10 device supports such presentation, the information will also be presented to these other 
people. After step 920, the subroutine continues to step 995 and returns. 

Those skilled in the an will appreciate that the selection of an output 
device and the formatting of the output information for that device can be performed in 
a variety of ways. For example, other user condition variables and description 
15 information factors could be used, or the selection could be made without reson to such 
information. For example, in one embodiment, the user could explicitly indicate the 
output device and formatting desired for some or all pieces of output information (e.g.. 
in response to an notification from the system), or another env.ty (e.g., an application 
program supplying the output information) could explicitly designate the output device 
20 and/or formatting. 

From the foregoing it will be appreciated that, although specific 
embodiments of the invention have been described herein for purposes of illustration, 
various modifications may be made without deviating from the spirit and scope of the 
invention. Accordingly, the invention is not limited except as by the appended claims. 
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CLAIMS 



1. A computer-implemented method for presenting output 
information to a user, the method comprising: 

receiving information about a modeled property of the user which affects 
appropriateness of presenting output information to the user; and 
presenting the output information to the user by, 

selecting an output device capable of presenting the output 
information in accordance with the modeled user property; and 

presenting the output information on the selected output device in 
accordance with the modeled user property 

2. The method of claim 1 wherein a plurality of output devices are 
available for presenting information to the user, wherein the modeled property of the 
user is one of a level of privacy desired by the user indicating a group of people allowed 
to perceive information presented by the computer, a scope of audience desired by the 
user indicating how many people are intended to perceive information presented by the 
computer, or a cognitive load of the user indicating ability of the user to devote attention 
to the computer, and including: 

(a) monitoring the user to collect information about a current state of the 

user; 

(b) modeling a current user condition based on the collected information 
by determining the current level of privacy desired by the user, the current scope of 
audience desired by the user, or the current cognitive load of the user; and 

(c) receiving the output information to be presented to the user, 

and wherein the selecting of the output device includes selecting one of the plurality of 
available output devices such that information presentation capabilities of the selected 
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16 output device support the determined current desired level of privacy, the determined 

17 current desired scope of audience, and the determined current cognitive load. 



1 3. The method of claim 2 wherein the modeling of the current user 

2 condition is additionally based on collected information related to a surrounding 

3 environment. 

1 4. The method of claim 2 wherein the monitoring of the user is 

2 performed by sensors worn by the user. 

1 5. The method of claim 2 wherein the monitoring of the user is 

2 performed by sensors remote from the user. 

1 6. The method of claim 2 wherein the collected information includes 

2 information about a current physiological state of the user 

1 7. The method of claim 2 wherein the collected information includes 

2 information about current activities of the user. 

1 8. The method of claim 2 wherein the computer is a wearable 

2 computer. 

1 9. The method of claim 2 wherein an operating system of the 

2 computer performs the method. 

1 10. The method of claim 2 wherein the output information is received 

2 from a distinct program executing on the computer. 

1 11. The method of claim 2 wherein the output information is received 

2 from another computer 
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1 12 The method of claim 2 wherein the output information is 

2 generated by the computer during the monitoring or the modeling. 

1 13 The method of claim 2 including receiving description information 

2 for the output information describing importance and deferability of the output 

3 information, and wherein the presenting of the output information to the user is 

4 additionally performed in a manner consistent with the description information. 

1 H The method of claim 2 wherein the presenting of the output 

2 information to the user is deferred when the output information cannot be presented on 

3 any of the available output devices in a manner consistent with the modeled current user 

4 condition. 

• 15. The method of claim 2 wherein the available output devices 

2 include two display devices with different information presentation capabilities regarding 

3 level of privacy, scope of audience, and cognitive load, and wherein the selected output 

4 device is the display device whose information presentation capabilities best match the 

5 modeled current user condition. 

1 16. The method of claim 2 wherein information can be presented with 

2 the available output devices to the user via at least two user senses, and wherein the 

3 selecting includes determining the user sense via which the output information will be 

4 presented. 

1 1 7. The method of claim 1 wherein the received information includes 

2 information related to a current condition of the user, and including modeling the user 

3 property based on the received information. 
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1 18 The method of claim 17 including monitoring the user to collect 

2 the received information. 

1 19. The method of claim 1 wherein the received information includes 

2 information related to a surrounding environment, and including modeling the user 

3 property based on the received information. 

1 20. The method of claim 19 including monitoring the surrounding 

2 environment to collect the received information. 

1 21. The method of claim 1 wherein the modeled user property is 

2 based on algorithmic modeling. 

1 22. The method of claim 1 including before the presenting of the 

2 output information, receiving the output information. 

1 23. The method of claim 22 wherein an information provider 



2 transmits information to various computers within a transmission range of the 

3 information provider, wherein the method is performed by a transportable computer 

4 transported by the user, wherein the user transports the transportable computer within 

5 the transmission range, and wherein the received output information is the transmitted 

6 information. 

1 24. The method of claim 1 including receiving description information 

2 indicating importance or deferability of the output information, and wherein the selecting 

3 is additionally performed in accordance with the description information. 
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1 25. The method of claim 1 including deferring the presenting of the 

2 output information when no selectable output device is capable of presenting the output 

3 information in accordance with the modeled user property. 

1 26 The method of claim 1 wherein the modeled user property is an 

2 indication of a degree of interruptibility of current activities of the user. 

1 27. The method of claim 26 wherein the degree of interruptibility of 

2 the current activities is influenced by information being received by the user via one user 

3 sense in conjunction with the current activities, and wherein the presenting of the output 

4 information is selected to be via a distinct user sense. 

1 28. The method of claim 26 including deferring the presenting of the 

2 output information when the degree of interruptibility of the current activities is low 

1 29. The method of claim 26 wherein the degree of interruptibility of 

2 the current activities is low when the current activities involve a risk of harm to the user. 

1 30. The method of claim 1 wherein the modeled user property is an 

2 indication of ability of the user to devote attention to the presenting of the output 

3 information. 

1 31. The method of claim 30 wherein the ability of the user to devote 

2 attention is influenced by information being received by the user via one user sense, and 

3 wherein the presenting of the output information is selected to be via a distinct user 

4 sense. 

1 32. The method of claim 30 including deferring the presenting of the 

2 output information when the ability of the user to devote attention is low. 
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1 33. The method of claim 30 wherein the indication of the ability of the 

2 user to devote attention is an estimate of an amount of attention devoted by the user to 

3 other current activities. 

1 34. The method of claim 1 wherein the modeled user property is a 

2 preference of the user for an amount of people to perceive presented information. 

1 35. The method of claim 34 wherein the selected output device 



2 includes a range of information presentation capabilities, wherein others can perceive 

3 information presented using some of the information presentation capabilities, and 

4 including selecting those information presentation capabilities to be used for the 

5 presenting when the modeled user property indicates that the others are within the 

6 amount of people to perceive the output information. 

1 36. The method of claim 34 wherein the selected output device 

2 includes a range of information presentation capabilities, wherein others nearby cannot 

3 perceive information presented using some of the information presentation capabilities, 

4 and including selecting those information presentation capabilities to be used for the 

5 presenting when the modeled user property indicates that the others exceed the amount 

6 of people to perceive the output information. 



1 37. The method of claim 1 wherein the modeled user property is a 

2 preference of the user for how to receive sensitive information 

1 38 The method of claim 37 wherein the selected output device 

2 includes a range of information presentation capabilities, wherein others can perceive 

3 information presented using some of the information presentation capabilities, and 

4 including selecting those information presentation capabilities to be used for the 
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5 presenting when the modeled user property indicates that sensitivity of the output 

6 information allows the others to perceive the output information. 



39. The method of claim 37 wherein the selected output device 
includes a range of information presentation capabilities, wherein others nearby cannot 
perceive information presented using some of the information presentation capabilities, 
and including selecting those information presentation capabilities to be used for the 

5 presenting when the output information is sufficiently sensitive that the modeled user 

6 property indicates that the others are not allowed to perceive the output information. 

1 40. The method of claim 1 wherein the modeled user property is a 

2 preference of the user for receiving information during current activities of the user. 

1 41. The method of claim 40 including deferring the presenting of the 

2 output information when the preference of the user is to not receive the output 

3 information during the current activities. 

1 42. The method of claim 40 including presenting the output 

2 information non-intrusively when the preference of the user is to not receive output 

3 information during the current activities. 

1 43. The method of claim 1 wherein the modeled user property is an 

2 indication of appropriateness of presenting the output information in a manner 

3 perceivable by others. 

1 44. The method of claim 43 wherein the selected output device 

2 includes a range of information presentation capabilities, wherein others cannot perceive 

3 information presented using some of the information presentation capabilities, and 

4 including selecting those information presentation capabilities to be used for the 
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5 presenting when the modeled user property indicates that presenting the output 

6 information in a manner perceivable by the others is not appropriate. 

1 45. The method of claim 43 including deferring the presenting of the 

2 output information when the modeled user property indicates that presenting the output 

3 information in a manner perceivable by others is not appropriate and available output 

4 devices cannot present the output information in a manner not perceivable by the others. 

1 46. The method of claim 1 wherein the selected output device 

2 includes a range of information presentation capabilities, and including formatting the 

3 output information before the presenting, the formatting to select information 

4 presentation capabilities of the selected output device to be used such that the selected 

5 inrormation presentation capabilities are nsistent with the modeled user property. 

1 47. The method of claim 1 wherein a plurality of output devices are 

2 available to present information via distinct senses of the user, and wherein the selecting 

3 of the output device includes determining a user sense which is capable of presenting the 

4 output information in accordance with the modeled user property. 

1 48. The method of claim 1 wherein the selected output device has 

2 only a single physical mechanism for presenting the output information to the user. 

1 49. The method of claim 1 wherein the computer performing the 

2 method is transportable by the user, wherein fixed output devices become available to 

3 the computer when the user transports the computer near the fixed output devices, and 

4 wherein the selected output device is a fixed output device. 

1 50 The method of claim 1 wherein a user computer performing the 

2 method is transportable by the user, and wherein the user computer can communicate 

3 with other devices within a transmission range of the user computer. 

BNSDOCID: <WO 0036493A1_I_> 



WO 00/36493 PCT/US99/29952 

40 



1 51. The method of claim 50 wherein the user computer communicates 

2 with another computer, and wherein the selected output device is an output device of the 

3 another computer 

1 52. The method of claim 50 wherein the received information is from 

2 one of the other devices. 

1 53. The method of claim 50 wherein the output information to be 

2 presented is received from one of the other devices. 

1 54. The method of claim 1 including after the presenting of the output 

2 information, revising the modeled user property based on the presenting. 

1 55. The method of claim 1 wherein the modeled user property 

2 indicates capabilities of the user for receiving presented output information. 

1 56. The method of claim 55 wherein the capabilities indicate that a 

2 physical disability of the user prevents the user from perceiving some types of 

3 presentations of information, and wherein the presenting of the output information is in a 

4 manner perceivable by the user. 

1 57. The method of claim 1 including modeling properties of the user 

2 which affect appropriateness of presenting output information to the user, the modeling 

3 including: 

■* receiving information about a current state of the user; and 

5 for at least one of a plurality of properties of the user, 

6 determining whether the received information relates to the 



7 property; and 
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8 when the received information relates to the property, determining 

9 a current value for the property based on the received information. 

1 58 The method of claim 57 including monitoring the user to obtain 

2 the received information. 

1 59 The method of claim 57 wherein multiple pieces of information 

2 are received which relate to a property of the user, wherein the multiple pieces of 

3 information are inconsistent as to the current value for the property, and wherein the 

4 determining of the current value involves mediating the inconsistencies. 

1 60. The method of claim 57 wherein a rating indicating quality of the 

2 received information is received, and wherein the quality rating is reflected in the 

3 determined current values for the properties which relate to the received information. 

1 61. The method of claim 1 wherein the computer is able to output 



2 information to a first display device and a second display device, the first and second 

3 display devices having different display characteristics, wherein the selecting of the 

4 output device includes selecting either the first display device or the second display 

5 device based on a predicted preference of the user, the predicted preference indicating 

6 on which of the display devices the user would prefer to receive the output information, 

7 and wherein the presenting of the output information on the selected output device in 

8 accordance with the modeled user property includes presenting the output information 

9 on the selected display device consistently with the predicted preference. 



1 62. The method of claim 61 including predicting a current value of the 

2 preference before the selecting. 

1 63. The method of claim 62 including monitoring the user to obtain 

2 information for the predicting. 
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1 64. The method of claim 61 wherein the predicted preference is a 

2 predicted mental state of the user, and wherein a mapping between the predicted mental 

3 state and the display devices is used for the selecting, the mapping indicating which of 

4 the display devices are suitable for presenting information in accordance with various 

5 predicted mental states. 



2 
3 



65. The method of claim 64 including: 

after the presenting, monitoring reaction of the user to the presenting; and 
revising the mapping based on the monitored reaction to enhance user 
reaction to future presentations of information. 



1 66. The method of claim 1 wherein the computer is a wearable 

2 computer worn by the user and has access to a plurality of output devices from which 

3 the user can receive information, wherein the modeled property of the user includes at 

4 least one of an ability of the user to currently receive the output information and a desire 

5 of the user of how to currently receive the output information, and including: 

6 monitoring the user to collect information; and 

7 characterizing the ability of the user to currently receive the output 

8 information and the desire of the user of how to currently receive the output information 

9 based on the collected information, 

10 and wherein the selecting of the output device includes selecting one of the plurality of 

1 1 output devices such that information presentation capabilities of the selected output 

12 device support the characterized ability and desire. 

1 67. The method of claim 66 including receiving description 

2 information indicating importance or deferability of the output information, and wherein 

3 the selecting is additionally performed in accordance with the description information 
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1 68. The method of claim 66 including receiving the output 

2 information from an application program executing on the wearable computer. 

1 69. A computer-readable medium containing instructions for 

2 presenting output information to a user by: 

3 receiving information about a modeled property of the user which affects 

4 appropriateness of presenting output information to the user; 

5 selecting an output device capable of presenting the output information in 

6 accordance with the modeled user property; and 

7 presenting the output information on the selected output device in 

8 accordance with the modeled user property. 

1 70. The computer-readable medium of claim 69 wherein the at least 

2 one modeled characteristic is based on collected information related to the user. 

1 71. The computer-readable medium of claim 70 wherein the computer 

2 system is further controlled by monitoring the user to obtain the collected information. 

1 72. The computer- readable medium of claim 69 wherein the computer 

2 system is further controlled by: 

3 receiving the output information; and 

4 receiving description information indicating importance or deferability of 



5 the output information, 

6 and wherein the selecting is additionally performed in accordance with the description 

7 information. 

1 73. The computer-readable medium of claim 69 wherein the selected 

2 output device includes a range of information presentation capabilities, and wherein the 

3 computer system is further controlled by formatting the output information before the 
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4 presenting, the formatting to select information presentation capabilities of the selected 

5 output device to be used such that the selected information presentation capabilities are 
consistent with the at least one modeled characteristic. 



6 



1 74 A computer system for presenting output to a user, comprising: 

2 an output device selector module that receives information about a 

3 modeled property of the user which affects appropriateness of presenting output to the 

4 user, that selects an output device capable of presenting the output in accordance with 

5 the modeled property, and that presents the output on the selected output device 

6 accordance with the modeled property 



3 



in 



1 75. The computer system of claim 74 further comprising the selected 

2 output device. 



1 76. The computer system of claim 74 further comprising a model of a 

2 current condition of the user, the model including a plurality of user properties including 

3 the modeled property. 

1 77. The computer system of claim 74 wherein the computer system 

2 further comprises a user characterization module that generates the modeled property 

3 based on collected information related to the user or to a surrounding environment. 

1 78 The computer system of claim 77 wherein the user 

2 characterization module further monitors the user or the surrounding environment to 

3 obtain the collected information 



1 79. The computer system of claim 74 wherein the output device 

2 selector module further receives description information indicating importance or 
deferability of the output, and wherein the selecting of the output device is additionally 



4 performed in accordance with the description information. 
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1 80. The computer system of claim 74 wherein the selected output 

2 device includes a range of information presentation capabilities, and further including a 

3 format module that formats the output before the presenting, the formatting to select 

4 information presentation capabilities of the selected output device to be used such that 

5 the selected information presentation capabilities are consistent with the modeled 

6 property. 



1 81 The computer system of claim 74 wherein the received 

2 information is current physiological information about the user indicating a health 

3 condition, and wherein the presenting of output is to another person to alert the another 

4 person of a health problem detected from the current physiological information. 

1 82. The computer system of claim 74 wherein the received 

2 information is current physiological information about the user indicating a health 

3 condition, and wherein the presenting of output is to administer medical care for the 

4 health condition. 

1 83 . A data structure stored in the memory of a computer for use in 

2 presenting output information to a user, the data structure containing modeled 

3 information regarding the user, the data structure used by: 

4 extracting information from the data structure including a current value of 

5 a modeled ability of the user to currently receive the output information or of a modeled 

6 desire of the user of how to currently receive the output information; 

7 selecting an output device capable of presenting the output information in 

8 accordance with the extracted information; and 

9 presenting the output information on the selected output device in 

10 accordance with the extracted information. 
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84. The data structure of claim 83 including a defined interface to 
receive updates to the modeled information from another computer. 

85. The data structure of claim 83 wherein the data structure further 
contains a plurality of modeling rules useful for determining based on received 
information about the user a current value for the modeled ability or desire. 

86. The data structure of claim 85 wherein the modeling rules are 
useful for mediating inconsistencies between multiple pieces of information which relate 
to the modeled ability or desire so as to determine the current value. 

87. A computer program adapted to model properties of a user for 
use when presenting output information to the user when the program is executed on a 
computer, the execution to perform a method comprising: 

receiving information about a current state of the user; and 
for at least one of a plurality of properties of the user which affects 
appropriateness of presenting output information to the user, modeling the property by 

determining whether the received information relates to the 

property; and 

when the received information relates to the property, determining 
a current value for the property based on the received information, 
so that the determined current values of the modeled user properties can be used for 
presenting output information to the user in an appropriate manner 

88. The computer program of claim 87 wherein the method includes: 
receiving output information to be presented in accordance with the 

determined current values of the modeled user properties; and 

presenting the output information to the user in accordance with the 

modeled user properties. 
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1 89 The computer program of claim 87 wherein the method includes 

2 when another program has output information to present to the user, supplying the 

3 determined current values of the modeled user properties to the another program so that 

4 the output information can be presented to the user by the another program in 

5 accordance with the modeled user properties. 

1 90. The computer program of claim 87 wherein the determining of the 

2 current value for the property is based on a plurality of modeling rules. 

1 91. The computer program of claim 90 wherein when the user is in a 

2 specified class of users, using modeling rules specialized for the specified class. 

1 92. The computer program of claim 90 wherein when the user is not 

2 in a pre-determined class of users, using default modeling rules. 

1 93. The computer program of claim 90 wherein the method includes 

2 updating the modeling rules based on the received information so as to better model the 

3 user. 

1 94. The computer program of claim 87 wherein the computer can 

2 communicate with another computer, the another computer able to obtain information 

3 about the user via input devices of the another computer, and wherein the received 

4 information is obtained information about the user from the another computer. 

1 95. The computer program of claim 87 wherein the computer can 

2 communicate with another computer, and including receiving from the another computer 

3 information about properties of the user to be added to the modeled user properties, the 

4 added properties distinct from the modeled user properties. 
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1 96 The computer program of claim 87 wherein the computer can 

2 communicate with another computer, and including receiving from the another computer 

3 a value for one of the modeled user properties to be stored as the determined current 

4 value for the one property. 

1 97. The computer program of claim 87 wherein the method includes 

2 monitoring the user to obtain the received information 



1 98. The computer program of claim 87 wherein multiple pieces of 

2 information are received which relate to a property of the user, wherein the multiple 

3 pieces of information are inconsistent as to the current value for the property, and 

4 wherein the determining of the current value involves mediating the inconsistencies. 

1 99. The computer program of claim 87 wherein a rating indicating 

2 quality of the received information is received, and wherein the quality rating is reflected 

3 in the determined current values for the properties which relate to the received 

4 information. 
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